Figure 1: Comparison between Lagrangian (left) and Eulerian (right) perspectives.

Recent approaches for modelling dynamics of physical systems with neural networks enforce Lagrangian or Hamiltonian structure to improve prediction and generalization. However, when coordinates are embedded in high-dimensional data such as images, these approaches either lose interpretability or can only be applied to one particular example. We introduce a new unsupervised neural network model that learns Lagrangian dynamics from images, with interpretability that benefits prediction and control. The model infers Lagrangian dynamics on generalized coordinates that are simultaneously learned with a coordinate-aware variational autoencoder (VAE). The VAE is designed to account for the geometry of physical systems composed of multiple rigid bodies in the plane. By inferring interpretable Lagrangian dynamics, the model learns physical system properties, such as kinetic and potential energy, which enables long-term prediction of dynamics in the image space and synthesis of energy-based controllers.

Figure 1: Comparison between Lagrangian (left) and Eulerian (right) perspectives.

Here we summarize all our model assumptions and highlight what are learned from data.

This is not obvious since recent work has applied Hamiltonian/Lagrangian to general latent spaces as opposed to rigid body systems.

This paper makes it possible to learn Lagrangian dynamics from images and use them for energy-based control. This represents an important and significant advance for this fledgling new research subfield of physics-aware prediction, which might very well go on to prove important and significant in the coming years. I believe the reviewers are all in agreement on this point. However, by entering this new territory for physics-aware prediction, this paper has also exposed itself to interest from a broader community of readers and NeurIPS attendees who are familiar with the progress in image-based *intuitive physics* modeling and control methods over the last 5 years or so (R2 and R4 point to some such approaches). A lot of the difficulty in arriving at a reviewer consensus for this paper can be put down to the fact that its positioning is somewhat myopic and ignores this broader context, perhaps because the authors themselves might not be familiar with these approaches.

Recent approaches for modelling dynamics of physical systems with neural networks enforce Lagrangian or Hamiltonian structure to improve prediction and generalization. However, when coordinates are embedded in high-dimensional data such as images, these approaches either lose interpretability or can only be applied to one particular example. We introduce a new unsupervised neural network model that learns Lagrangian dynamics from images, with interpretability that benefits prediction and control. The model infers Lagrangian dynamics on generalized coordinates that are simultaneously learned with a coordinate-aware variational autoencoder (VAE). The VAE is designed to account for the geometry of physical systems composed of multiple rigid bodies in the plane. By inferring interpretable Lagrangian dynamics, the model learns physical system properties, such as kinetic and potential energy, which enables long-term prediction of dynamics in the image space and synthesis of energy-based controllers.